Methods and apparatus for voice-enabling a web application

ABSTRACT

Methods and apparatus for voice-enabling a web application, wherein the web application includes one or more web pages rendered by a web browser on a computer. At least one information source external to the web application is queried to determine whether information describing a set of one or more supported voice interactions for the web application is available, and in response to determining that the information is available, the information is retrieved from the at least one information source. Voice input for the web application is then enabled based on the retrieved information.

BACKGROUND

Many computer applications are made available for use or access over theInternet. Such web applications typically execute, at least in part, ona client computer and are rendered in a web browser of the clientcomputer, with program logic being executed by some combination of theweb browser and one or more web servers hosting the application. Webapplications typically use application logic coded in abrowser-supported language (e.g., JavaScript) combined with abrowser-rendered markup language (e.g., HTML). More recent developmentsin web application design have been directed at providing webapplications using technologies such as AJAX, which allow for increaseduser interactivity without having to reload entire web page for the webapplication each time a user interacts with the web page. Such dynamicupdates to web pages enables faster operation of a web application byreducing the amount of data transfer and rendering required in responseto user interactions.

SUMMARY

Some embodiments are directed to a method of voice-enabling a webapplication, wherein the web application includes one or more web pagesrendered by a web browser, the method comprising: querying at least oneinformation source external to the web application to determine whetherinformation describing a set of one or more supported voice interactionsfor the web application is available, wherein the at least oneinformation source includes voice interaction information for aplurality of web applications; retrieving from the at least oneinformation source, in response to determining that the informationdescribing a set of one or more supported voice interactions for the webapplication is available, the information describing the set ofsupported voice interactions for the web application; and enabling voiceinput for the web application based on the retrieved informationspecifying the set of one or more supported voice interactions.

Some embodiments are directed to a computer-readable storage mediumencoded with a plurality of instructions that, when executed by acomputer, perform a method comprising: querying at least one informationsource external to the web application to determine whether informationdescribing a set of one or more supported voice interactions for the webapplication is available, wherein the at least one information sourceincludes voice interaction information for a plurality of webapplications; retrieving from the at least one information source, inresponse to determining that the information describing a set of one ormore supported voice interactions for the web application is available,the information describing the set of supported voice interactions forthe web application; and enabling voice input for the web applicationbased on the retrieved information specifying the set of one or moresupported voice interactions.

Some embodiments are directed to a computer comprising: at least oneprocessor programmed to: query at least one information source externalto the web application to determine whether information describing a setof one or more supported voice interactions for the web application isavailable, wherein the at least one information source includes voiceinteraction information for a plurality of web applications; retrievefrom the at least one information source, in response to determiningthat the information describing a set of one or more supported voiceinteractions for the web application is available, the informationdescribing the set of supported voice interactions for the webapplication; and enable voice input for the web application based on theretrieved information specifying the set of one or more supported voiceinteractions.

Some embodiments are directed to at least one server computer,comprising: at least one storage device storing a set of informationthat specifies a set of voice interactions for at least one webapplication; and a communication interface configured to: receive from aclient computer, via at least one network, a request for at least someof the set of information; and provide, via the at least one network,the at least some of the set of information to the client computer inresponse to the request.

Some embodiments are directed to a computer-readable storage mediumencoded with a plurality of instructions that, when executed by at leastone server computer, perform a method comprising: receiving from aclient computer, via at least one network, a request for informationthat specifies a set of voice interactions for at least one webapplication; and providing, via the at least one network, theinformation to the client computer in response to the request.

Some embodiments are directed to a method, comprising: receiving from aclient computer, via at least one network, a request for informationthat specifies a set of voice interactions for at least one webapplication; and providing, via the at least one network, theinformation to the client computer in response to the request.

Some embodiments are directed to a method of determining a collectiveset of supported voice interactions for a plurality of frames in a webbrowser including a first frame and a second frame, wherein the firstframe corresponds to a first web application and the second framecorresponds to a second web application, the method comprising:determining a first set of supported voice interactions available forthe first frame; determining a second set of supported voiceinteractions available for the second frame; determining the collectiveset of supported voice interactions based on the first set of supportedvoice interactions and the second set of voice interactions; andinstructing an external speech engine to recognize voice inputcorresponding to the collective set of voice interactions.

Some embodiments are directed to a computer-readable storage mediumencoded with a plurality of instructions that, when executed by acomputer, perform a method of determining a collective set of supportedvoice interactions for a plurality of frames in a web browser includinga first frame and a second frame, wherein the first frame corresponds toa first web application and the second frame corresponds to a second webapplication, the method comprising: determining a first set of supportedvoice interactions available for the first frame; determining a secondset of supported voice interactions available for the second frame;determining the collective set of supported voice interactions based onthe first set of supported voice interactions and the second set ofvoice interactions; and instructing an external speech engine torecognize voice input corresponding to the collective set of voiceinteractions.

Some embodiments are directed to a computer comprising: at least oneprocessor programmed to perform a method of determining a collective setof supported voice interactions for a plurality of frames in a webbrowser including a first frame and a second frame, wherein the firstframe corresponds to a first web application and the second framecorresponds to a second web application, the method comprising:determining a first set of supported voice interactions available forthe first frame; determining a second set of supported voiceinteractions available for the second frame; determining the collectiveset of supported voice interactions based on the first set of supportedvoice interactions and the second set of voice interactions; andinstructing an external speech engine to recognize voice inputcorresponding to the collective set of voice interactions.

Some embodiments are directed to a method of enabling voice interactionfor at least one capability of a web application including one or moreweb pages rendered by a web browser, wherein the at least one capabilityis not exposed by the web browser, the method comprising: executing anagent for the web application, wherein the agent is configured todetermine an identity of the web application; monitoring events in theweb browser to identify at least one marker identifying informationabout a current context of the web application; and enabling voiceinteraction for the at least one capability associated with the currentcontext in response to identifying the at least one marker, wherein theat least one capability is not exposed by the web browser.

Some embodiments are directed to a computer-readable storage mediumencoded with a plurality of instructions that, when executed by acomputer, perform a method of enabling voice interaction for at leastone capability of a web application including one or more web pagesrendered by a web browser, wherein the at least one capability is notexposed by the web browser, the method comprising: executing an agentfor the web application, wherein the agent is configured to determine anidentity of the web application; monitoring events in the web browser toidentify at least one marker identifying information about a currentcontext of the web application; and enabling voice interaction for theat least one capability associated with the current context in responseto identifying the at least one marker, wherein the at least onecapability is not exposed by the web browser.

Some embodiments are directed to a computer comprising: at least oneprocessor programmed to perform a method of enabling voice interactionfor at least one capability of a web application including one or moreweb pages rendered by a web browser, wherein the at least one capabilityis not exposed by the web browser, the method comprising: executing anagent for the web application, wherein the agent is configured todetermine an identity of the web application; monitoring events in theweb browser to identify at least one marker identifying informationabout a current context of the web application; and enabling voiceinteraction for the at least one capability associated with the currentcontext in response to identifying the at least one marker, wherein theat least one capability is not exposed by the web browser.

Some embodiments are directed to a method of enabling voice interactionfor invoking at least one capability of a web application including atleast one web page rendered by a web browser, the method comprising:analyzing a document object model of the at least one web page toidentify one or more items in the document object model at a first pointin time; determining based, at least in part, on the identified one ormore items, that the at least one web page comprises the at least onecapability; and enabling voice input to invoke the at least onecapability of the web application in response to the identifying thatthe at least one web page comprises the at least one capability.

Some embodiments are directed to a computer-readable storage mediumencoded with a plurality of instructions that, when executed by acomputer, perform a method of enabling voice interaction for invoking atleast one capability of a web application including at least one webpage rendered by a web browser, the method comprising: analyzing adocument object model of the at least one web page to identify one ormore items in the document object model at a first point in time;determining based, at least in part, on the identified one or moreitems, that the at least one web page comprises the at least onecapability; and enabling voice input to invoke the at least onecapability of the web application in response to the identifying thatthe at least one web page comprises the at least one capability.

Some embodiments are directed to a computer comprising: at least oneprocessor programmed to perform a method of enabling voice interactionfor invoking at least one capability of a web application including atleast one web page rendered by a web browser, the method comprising:analyzing a document object model of the at least one web page toidentify one or more items in the document object model at a first pointin time; determining based, at least in part, on the identified one ormore items, that the at least one web page comprises the at least onecapability; and enabling voice input to invoke the at least onecapability of the web application in response to the identifying thatthe at least one web page comprises the at least one capability.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided thatsuch concepts are not mutually inconsistent) are contemplated as beingpart of the inventive subject matter disclosed herein.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is an exemplary system including components used to voice-enablea web application executing in a browser on a client computer inaccordance with some embodiments of the invention;

FIG. 2 shows an illustrative process for enabling voice interaction(s)for a web application in accordance with some embodiments of theinvention;

FIG. 3 shows an illustrative process for enabling voice input for a webpage of a web application based, at least in part, on a context of theweb application in accordance with some embodiments of the invention;

FIG. 4 is a schematic illustration of a web application including aplurality of markers, one or more of which may be used in determining acontext of the web application in accordance with some embodiments ofthe invention;

FIG. 5 is a schematic illustration of a voice-enabled web page inaccordance with some embodiments of the invention, wherein the web pageincludes a plurality of frames associated with different sets of voiceinteractions defined for a web application associated with therespective frame;

FIG. 6 shows an illustrative process for determining a set of collectivesupported voice interactions when at least two web applications arepresent in a single web page in accordance with some embodiments of theinvention;

FIG. 7 shows an illustrative process for providing informationspecifying a set of voice interactions from a web-accessible repositoryto a client computer in response to a request in accordance with someembodiments of the invention; and

FIG. 8 is an illustrative client computer system that may be used inconnection with some embodiments of the invention.

DETAILED DESCRIPTION

The inventors have recognized and appreciated that despite the recentwidespread adoption of speech recognition technologies, many webapplications lack native voice-interface functionality. Because many webapplications as developed do not include voice capabilities, a browseron a client computer, which is used to render such applications does notexpose any voice capabilities of the web application. Some embodimentsof the invention are directed to methods and apparatus forvoice-enabling capabilities of a web application that have been designedwithout native voice interface capabilities.

An illustrative system 100 for use in accordance with some embodimentsof the invention is shown in FIG. 1. The system 100 includes a clientcomputer 110 configured to display a web page 121 of web application 120rendered by a browser 112. Web application 120 may include one or moreweb pages 121, each of which includes one or more elements to bedisplayed on the web page when rendered by browser 112. When browser 112renders a web page 121 of web application 120, the browser parses themarkup (e.g., HTML) for the web page and creates a Document Object Model(DOM) 114 from it. The DOM 114 is then used by the browser, optionallyin combination with one or more Cascading Style Sheets (CSS) to displaythe web page 121 in a browser window on client computer 110. Some webpages (but not all) use a CSS, which describes presentation semantics ofa document, in combination with markup (e.g., HTML) elements because itallows for the separation of web page content (e.g., as defined by theHTML elements), and web page presentation style and format as defined inthe CSS. Accordingly, the same markup content can be interpreted anddisplayed by the browser in different ways depending on the particularCSS that is used.

Some web pages are “static,” in the sense that each user interactionwith the web page would requires the entire page (or a new page) bere-loaded from an external server to reflect the update. As discussedabove, more recent advances in web application development allow for webpages with rich content and more efficient client-side interactivity byallowing for “dynamic” update of web pages for a web application. Suchcontent rich web applications (often called “rich internet applications”or “RIAs”) typically use advanced application logic, such as AJAX orAdobe Flash, to implement the dynamic behavior. Web applicationsdeveloped using technologies such as AJAX enable the web application toasynchronously communicate with an external server to send and receiveinformation without interfering with the current state (e.g., display orbehavior) of the page being displayed. The information received from theserver(s) is used to update the current web page without requiring afull page reload. For example, if web application 120 is developed usingAJAX, the web application may send and receive content to/from webapplication server 150 via network 160 without interfering with the webpage currently displayed in the browser window.

Using advanced application logic such as AJAX for developing webapplications also enables real-time updates of data on web pages, whichprovides the user with a more interactive experience. For example, afinancial web application may include one or more web pages with areal-time stock ticker that may be updated by sending requests to andreceiving data from one or more web application servers that source thedata. Methods of communicating with web application server(s) in thebackground, such as using XMLHttpRequest JavaScript objects, are known.

Many web applications include scripts written in a scripting languagesuch as JavaScript that execute within browser 112 and which provide aninteractive user interface for a web page of the web application.Scripting functions interact with the DOM of a web page to providefunctionality such as animation of page elements and interactivecontent. When used in combination with advanced application logic suchas AJAX, scripting functions additionally enable data and/or requestsfor information to be sent to a server asynchronously, as discussedabove.

In some embodiments, client computer 110 may also include a voiceapplication 130 configured to voice enable at least one capability ofweb application 120, even though web application 120 may not have beendeveloped with a native voice interface. Voice application 130 may beany suitable voice recognition application including, but not limitedto, a general speech recognition application that also provides a voiceinterface for other applications executing on the client computer 110,such as Dragon™ NaturallySpeaking® or a dictation application such asDragon™ Dictate, both of which are available from Nuance Communications,Inc.

Client computer 110 may include a voice interface 132 configured toreceive voice input from one or more voice input devices such as amicrophone 134. When a user speaks into microphone 134, the microphonemay transform input acoustic energy corresponding to the user's speechinto an electrical information which is sent as voice input to anautomatic speech recognition (ASR) engine 136 via voice interface 132.The voice interface 132 may perform one or more additional functionsprior to sending the voice input to ASR engine 136 including, but notlimited to, analog-to-digital conversion. ASR engine 136 receives thevoice input from the voice interface 132 and performs speech recognitionon the voice input to recognize the content of the received voice input.

Although ASR engine 136 is illustrated in FIG. 1 as being included as aportion of client computer 110, ASR engine 136 may alternatively belocated remotely from client computer 110 and may be in communicationwith client computer 110 using any suitable communication medium ormediums (e.g., one or more networks). In some embodiments, clientcomputer 110 may be associated with one or more local ASR engines 136and one or more remotely located ASR engines, whereby at least somereceived voice input is processed by the one or more local ASR engine(s)and at least some received voice input is processed by the one or moreremotely-located ASR engines. Examples of such distributed ASR systemsare known. Voice input sent to the one or more remote ASR engine(s) forrecognition using one or more communication medium(s) (e.g., one or morenetworks) connecting the client computer 110 and the ASR engine(s) maybe returned to the client computer 110 from the remotely-located ASRengine(s) using the same or different communication medium(s). Voiceinput recognized by ASR engine 136 (and/or remote ASR engine(s)) isprovided to voice application 130 and the recognized voice input may beused to control at least one capability of web application 120, asdiscussed in more detail below.

Many browsers include APIs to query the objects in the DOM to allow arequesting application visibility into what was rendered by the browser.In web applications that use static web pages that are fully reloaded inresponse to a user interaction, voice application 130 may send a requestto browser 112 and the browser may use one or more APIs to learn thecontent of DOM 114 for the particular web page that is being rendered bybrowser 112. Under these circumstances, because there is no dynamicupdate of the content of the web page, the DOM 114 contains all of theinformation about each of the elements of the web page, as each isspecified in the markup (e.g., HTML) for the web page, which was parsedby the browser when the page was loaded. The voice application may beprogrammed to include instructions to voice enable the discoverableelements in the DOM, e.g., by associating particular voice input with anaction to perform on a particular HTML element.

As discussed above, some web applications use advanced application logicto program (e.g., with Javascript) one or more visual elements in theDOM to respond to user interactions (e.g., mouse clicks, keystrokes,etc.). When visual element(s) of the DOM are programmed in such a way,inspection of the contents of the DOM is insufficient to determine thenature and/or purpose of the programmed element(s) as the detail aboutthe content of the element(s) is not ascertainable from the DOM.Moreover, without the ability to determine which DOM elements are userinterface elements and what their behavior should be, it is difficult todetermine which elements to speech-enable and how to speech-enable them.

The inventors have recognized and appreciated that it is a challenge toenable voice support for web applications that include web pages withelements that are not identifiable from an inspection of the contents ofthe DOM. Some embodiments are directed to providing techniques to voiceenable such web pages.

In accordance with a non-limiting example, a conduit for informationbetween voice application 130 and an agent 116 (e.g., a plugin)executing inside browser 112 is provided. The agent 116 is configured toaccess the browser's environment to facilitate voice enablement of atleast one capability of the web application presented by the browser. Inone embodiment, the agent 116 comprises one or more modules configuredto facilitate a loading of the agent by the browser for a web page thatis displayed (e.g., by using extension 122) and to access local storageand to communicate with a speech engine. It should be appreciated thatthe agent 116 may be implemented to enable any other functionality asembodiments are not limited in this respect.

In some embodiments, the extension 122 may be customized for aparticular type of browser (e.g., Internet Explorer, Firefox, Safari,etc.), and/or a particular operating system platform (e.g., MicrosoftWindows, Mac OSX, Linux, etc.) executing on computer 110. In suchembodiments, extension 122 may be configured to be both operatingsystem- and browser-specific because browsers are often customized tointeract with components of each operating system differently. However,the particular manner in which extension 122 is implemented does notlimit embodiments of the invention.

To prevent security vulnerabilities to viruses, etc., browser securitymodules typically restrict access to operating system APIs (e.g., filesystem, network) to agent(s) (e.g., plugins) that are executing for aparticular web page. Because agents are given access to operating systemAPIs, browsers prohibit plugins from automatically loading themselvesinto any arbitrary web page. Rather, an agent must be loaded after a webpage has been rendered by the browser. In one non-limiting exampledescribed herein, extension 122 is configured to load agent 116 inresponse to determining that a web page has been rendered by thebrowser. Because the nature of web page elements is not readilyascertainable from the DOM for some web applications (e.g., richinternet applications), agent 116, once loaded, executes within thebrowser to monitor the web application 120 to identify currentcontext(s) of the application and provide information to voiceapplication 130 when the context changes. Information about the currentcontext(s) of the application may be provided to voice application 130in any suitable way, as embodiments are not limited in this respect.

In some embodiments, client computer 110 may include a plurality ofbrowsers 112, and each browser may be associated with a differentextension 122 that is configured for use with a particular browser.However, all embodiments are not limited to this, and some embodimentsmay include one universal extension configured to work with allbrowsers.

As discussed above, in some embodiments, when a browser is launched onclient computer 110, and each time a new web page is loaded, extension122 may facilitate the loading of the agent 116 for the new web page. Asa result, as discussed in more detail below, when a set of one or moresupported voice interactions for a particular web application isavailable, agent 116 executing within the browser, monitors events forparticular web application pages. Because the extension 122 may beconfigured to execute whenever its associated browser 112 is launched,extension 122 is not given access to restricted operating system APIs toprevent security vulnerabilities. For example, in some embodiments, thefunctionality of the extension 122 may be restricted to activities suchas loading the agent 116 in the browser 112 when a new web page isrendered by the browser.

In some embodiments, agent 116 (e.g., a plugin) may comprise a set ofcomputer-executable instructions that, when executed within browser 112,provide additional functionality to web applications executing in thebrowser. For example, agent 116 may facilitate the discovery ofinformation about a state or context of a web page of the webapplication, which is not readily discernible from querying the DOM ofthe browser and this information can be used in any suitable way. Forexample, in one embodiment, the information may be used to enable voicecapabilities not native to the web application 120, although theinformation may alternatively be used in some other way. Agent 116 maybe a natively compiled module (i.e., a plugin) programmed in accordancewith the NPAPI standard, or agent 116 may be any other suitable set ofcomputer-executable instructions that implement a browser plugin API, asthe aspects of the disclosure relating to the use of a plugin are notlimited to any particular implementation.

In some embodiments, agent 116 may be configured to gather informationabout aspects of the web application 120 on behalf of voice application130 to determine whether and which capabilities of the web applicationcan be voice enabled. In addition, the agent may be configured to gatherinformation about each web page to determine which (if any) of thecommands supported by an application are relevant and available for eachparticular web page. In accordance with some embodiments, agent 116 maybe configured to determine a current context of a web application orapplications by monitoring one or more markers active in the web pagedisplayed by the application(s). Rather than directly identifyingelements from the DOM, agent 116 may be configured to use DOM events(e.g., a load event) to discover information about the HTML elements ofthe page itself to determine the current context for the webapplication. For example, the web application may be an emailapplication such as Gmail™ provided by Google, Inc. or MicrosoftHotmail® provided by Microsoft Corporation. Each of these email webapplications may include a plurality of web pages focused on differentaspects of the email application. For example, an email application mayinclude an “Inbox” web page, a “Compose message” web page, and a “Spam”web page. Each of these web pages is associated with a different contextof the application, and may have different elements, commands, and/oractions that may be voice enabled in accordance with some embodiments.Agent 116 may use one or more browser DOM APIs to monitor for one ormore markers to determine and identify the current context of the webapplication. For example, agent 116 may monitor for a “Send” userinterface (UI) element to be displayed on the web page and in responseto determining that the “Send” UI element is present, the agent maydetermine that the “Compose message” context is active and sendinformation regarding this context to voice application 130, therebyenabling the voice application to make active the voice commandsrelevant to that context. Although this example describes monitoring fora particular UI element to determine the current context, it should beappreciated that agent 116 may employ any suitable technique fordetermining the context, and is not limited to monitoring for aparticular UI element on the page to determine context.

In some embodiments, agent 116 may be configured to determine anidentity of a web application in response to a web page being loaded bybrowser 112. The identity of a web application may be determined in anysuitable way. As one non-limiting example, the identity of a webapplication may be determined based, at least in part, on an identifierfor the web page loaded by the browser. For example, the identifier forthe web page may be the Uniform Resource Locator (URL) associated withthe web page, distinctive content displayed on the web page (e.g., theGmail™ logo), or some other identifier. In response to determining theidentity of a newly-loaded web application, agent 116 may determinewhether the web application is associated with a set of one or morevoice interactions that provide a voice interface for the webapplication. This can be done in any suitable way. For example, in oneembodiment, the agent 116 may have information in a data structurestored locally on the same computer on which the agent is installed andthat identifies applications for which the agent supports voiceinteraction and identifies for each application, the particular voicecommands that are supported. Thus, each agent installed on a computermay have its own data structure with information that identifies whatapplications and commands are voice enabled. However, in otherembodiments, the information may be stored remotely from the agent andthe computer on which the agent is installed, in a manner that makes theinformation globally accessible (e.g., over the Internet or anothernetwork) to multiple agents on different computers. This enables updatesto the information to be made and quickly and easily distributed tomultiple agents (which can include thousands or more), without having toindividually update separate data structures maintained for each agent.

In some embodiments, the set of one or more voice interactions for a webapplication are specified in a data structure called a ContextualCommand Sheet (CCS), as described in more detail below. In embodimentsthat use a CCS, agent 116 may issue a query to the CCS (e.g., which mayinclude a local CCS cache 170 and globally-accessible CCS repository 180as shown in FIG. 1 and described below) to determine whether a CCS forthe identified web application is available. A CCS data structure may bea simple document or take any other suitable form and examples of CCSsfor specifying a set of one or more voice interactions for a webapplication are described in more detail below.

FIG. 2 shows an illustrative process for enabling voice input for a webapplication in accordance with some embodiments. In act 210, one or moreinformation sources external to a web application (e.g., a CCS) arequeried by an agent executing in a web browser to determine whetherinformation describing a set of one or more supported voice interactionsfor the web application is available. As discussed above, in someembodiments, prior to querying the information source(s), the identityof the web application may be determined in any suitable way (e.g.,using the URL of the web page currently loaded in the browser). The oneor more information sources may then be queried by sending a requestthat includes the identity of the web application to the informationsource(s).

Some embodiments may include an information source such as a collectionof CCSs (e.g., a manifest which can be stored in a single document,multiple documents, or any other suitable data structure) stored at alocation accessible to the client computer 110 locally or over one ormore networks. For example, the information may be located on apublically-accessible server (e.g., located at the URLhttp://nuance.com) configured to receive requests for information. Insome embodiments, the information stored on the publically-accessibleserver may include the identities of all web applications for whichthere is a corresponding CCS, where each CCS defines the set ofsupported voice interactions for the corresponding web application. Insome embodiments, in response to consulting a manifest file storedlocally, a request for a remotely-stored CCS may be sent from the clientcomputer to the server, wherein the request includes a unique identifierfor the web application.

In response to receiving the request, the server may determine whetherthe web application identified in the request is associated with voiceinteraction information. If not, the process proceeds to act 240 where aresponse to the request is sent (e.g., to the requesting agent) toindicate that the web application cannot be voice enabled. If it isdetermined that there is voice interaction information for the webapplication, the process proceeds to act 220, where the voiceinteraction information is retrieved. The voice interaction informationmay be retrieved in any suitable way from any suitable informationsource. In one embodiments, as shown in FIG. 1, client computer 110 mayinclude a local data source such as CCS cache 170 configured to storeone or more CCSs that have already been retrieved from an externalinformation source such as CCS repository 180. Providing some CCSslocally in a cache provides an advantage of quicker access by notrequiring access to a remotely-located information source, however,embodiments are not limited to using a cache and the CCSs may be storedin any location accessible to a requesting agent. In some embodiments,information (e.g., a CCS) including voice interaction information for aweb application may be associated with a particular version of thatinformation (e.g., a CCS version), and the CCS cache 170 may be queriedto determine whether the version of the CCS in the CCS cache 170 for theloaded application is the most recent version. In one embodiments, if itis determined that a more recent version is available, the more recentversion of the CCS may be retrieved from the external information source(e.g., CCS repository 180). In other embodiments, the externalinformation source may store information about which agents have whichCCSs and if the version of the CCS is updated, the updated version ofthe CCS may be pushed to computers with older versions of the CCS. Inyet other embodiments, when a CCS is updated to a new version on theexternal information source, a notice may be sent to all agents witholder cached versions of the CCS to flush the cached copy so the newversion will be used the next time the corresponding web application islaunched. The current version of the CCS may be accessed and providedlocally to a requesting agent in any suitable way and the techniquesdescribed above are provided for merely illustrative purposes.

If it is determined that there is no available CCS (either in the CCScache 170 or at an external information source) for the web applicationloaded in the browser, it may be determined that there is no voicesupport for the application and execution of the web application mayproceed normally. Otherwise, if it is determined that a CCS is availablefor the loaded web application, the process proceeds to act 230, wherethe web application is enabled for voice input based, at least in part,on the voice interaction information defined in the CCS.

Voice enabling the web application may be performed in any suitable way.In some embodiments, agent 116 executing inside of browser 112 mayinstruct voice application 130 to dynamically configure its grammar forrecognizing input speech based, at least in part, on a current contextof the web application and the voice commands that the CCS indicates areenabled for that web application. For example, the CCS may specify thefull set of voice enabled commands for the application, and the agentmay determine the current context and only instruct the voiceapplication to enable the commands relevant for the context. It shouldbe appreciated that all embodiments are not limited to constraining thevoice enabled commands to the relevant context, but that this is done insome embodiments to attempt to improve the accuracy of the voicerecognition by constraining it to choosing between only commands foractions relevant to the context.

In some embodiments, each CCS may be a document (e.g., an XML document)or some other data structure that includes information mapping speechinput (e.g., voice commands) to particular elements of the webapplication. An illustrative portion of a CCS for the web applicationGmail™ using XML is shown below.

    <?xml version=“1.0” encoding=“UTF-8”?> <ccs>   <id>gmail</id>  <name>GMail</name>   <urlmarker>google.com/mail/</urlmarker>  <version>1</version>   <lang>en</lang>   <context id=“base”>    <description>Global</description>     <cmdname=“showwebsitecomands”>       <sample>Show [web] sitecommands</sample>       <trigger>Show {website|site} commands</trigger>      <invokeScript> nuan_whatCanISay( ); </invokeScript>     </cmd>    <cmd name=“compose”>       <sample>[click:Click] Compose | Composenew       email</sample>       <marker>.z0 div</marker>      <trigger>{[click:Click] Compose|Compose new       email</trigger>      <invokeScript> nuan_el = nuan_getElement(marker);        nuan_mousedown(nuan_el);         nuan_mouseup(nuan_el);      </invokeScript>     </cmd>     <cmd name=“inbox”>      <sample>[click:Click] Inbox, | Go to Inbox</sample>      <marker>a.J-Ke[href$=inbox]</marker>       <trigger>{[click:Click]|Go To} Inbox</trigger>       <invokeScript> var el =nuan_getElement(marker);         nuan_click(el); </invokeScript>    </cmd>   </context>   <context id=“compose”>    <description>Composing Message</description>    <marker>.T-I.J-J5-Ji.Bq.nS:visible</marker><!-- send button -->    <cmd name=“gototo”>       <sample>Click To | Go to To</sample>      <trigger>{Click|Go to} To </trigger>       <invokeScript>        nuan_getElement(‘textarea[name=to]:visible’).-         focus( );      </invokeScript>     </cmd>     <cmd name=“gotosubject”>      <sample>[click:Click] Subject | Go to Subject</sample>      <trigger> {[click:Click]|Go to} Subject</trigger>      <invokeScript>        nuan_getElement(‘input[name=subject]:visible’).-         focus();       </invokeScript>     </cmd>     <cmd name=“gotobody”>      <sample>[click:Click] Body | Go to Body</sample>     <trigger>{[click:Click]|Go to} Body </trigger>     <invokeScript>    nuan_getElement(‘body.editable:visible’).focus( );    </invokeScript>   </cmd>   <cmd name=“discard”>    <sample>[click:Click] Discard | Discard Message</sample>    <marker>.T-I.J-J5-Ji.Bq.lX.T-I-ax7.L3</marker>     <trigger>{[click:Click] Discard|Discard message}</trigger>     <invokeScript>    var nuan_el = nuan_getElement(marker);      nuan_mousedown(nuan_el); nuan_mouseup(nuan_el);    </invokeScript>   </cmd> </context> </ccs>

As can be appreciated from the illustrative CCS above, a CCS defines aset of voice interactions (e.g., “Discard Message,” “Go to Subject”)that correspond to voice commands that a user may invoke to control thefunctionality of the email application. In some embodiments, the CCS maydefine during which context each of the voice interactions in the set isactive, although not all embodiments are limited in this respect. Theset of voice interactions defined in any particular CCS may include anysuitable commands for controlling any capabilities of a web applicationthat the application natively provides the ability to control (albeitwithout voice). In this manner, rich command and control of theapplication by voice can be provided to create a rich voice experience,without requiring web application designers to incorporate a voiceinterface during design of the application.

In some embodiments, as described above, agent 116 may interpret a CCSto determine which voice interactions to enable and which to disablebased, at least in part, on a current context of the web application.FIG. 3 shows an illustrative process for determining which of the set ofvoice interactions to enable/disable in accordance with someembodiments. In act 310, an agent (e.g., NPAPI plugin) executing in thebrowser analyzes one or more elements of the currently loaded web pageto determine the context of the web application. For example, asdescribed above, in some embodiments, the agent may handle DOM events totrigger the execution code that analyzes whether a particular marker ispresent on the web page, and based on which marker(s) are present, thecurrent context may be determined. The process then proceeds to act 320,where it is determined whether the web page includes a capability to bevoice enabled. For example, FIG. 4 shows an illustrative web page 400 ofan email web application rendered in browser 412. In response todetermining that the web application has a corresponding CCS, the agent416 has loaded the CCS and proceeds to monitor for one or more markersspecified in the CCS that indicate the current context of webapplication. For example, the CCS may include information that statesthat when a send UI element 414 is present the corresponding context forthe web application is “Compose Message.” Because a send UI element 414is present on the currently rendered web page 400 as shown in FIG. 4,the agent 416 may determine that the context is “Compose Message,” andthe agent may query the CCS to determine which voice interactions shouldbe enabled for this context.

After determining which voice interactions should be enabled, theprocess of FIG. 3 proceeds to act 330, where voice input for the currentcontext is enabled to invoke the appropriate voice capabilities asspecified in the CCS. Although the determination of context in theabove-described example is made by detecting only a single marker (i.e.,send UI element 414), the CCS may alternatively specify that detectionof more than one marker (any suitable number) be used to determine aparticular context, as embodiments are limited in this respect. Forexample, a determination of a particular context may be specified notonly on detection of send UI element 414, but also on detection ofmessage text box 418 and/or compose message toolbar 420.

Voice enabling at least some portions of the web application inaccordance with the information in a CCS may be accomplished in anysuitable way. In one embodiment, the CCS for the web application mayindicate a set of voice commands to activate and the ASR may beconfigured to recognize the voice commands in the set. For example, theset of voice commands may include the command “Send” corresponding to anaction of sending an email message in a web application such as Gmail™.In response to identifying user input corresponding to the command“Send,” the voice application may provide information to the agentexecuting in the browser to inform the agent that “Send” voice commandhas been invoked. This information may be transferred from the voiceapplication to the agent in any suitable way, such as using one or moreAPIs. After the agent has received the information indicating that theuser has invoked a voice command, the agent may use the CCS to determinewhich action to perform that corresponds to the input voice command andthe agent may use one or more APIs to instruct the web application toperform the one or more actions associated with the voice command.

The CCS may specify a link between the voice command and a correspondingaction to perform in any suitable way. In one embodiment, all of theinformation need to invoke a particular action may be included in theCCS itself. In other embodiments, information in the CCS that links avoice command to one or more actions to be perform may be specifiedusing one or more links to one or more functions (e.g., a JavaScriptfile, a library file) having computer-executable instructions locatedsomewhere other than the CCS. Providing some functionality fordescribing action(s) to associate with voice commands in locations otherthan the CCS may be advantageous in some circumstances, such as when theaction to be performed includes complex behavior, which may include manyinstructions (e.g., to instruct APIs) to perform to accomplish thecomplex behavior. The particular way in which instructions arespecified, either in the CCS itself, or in one or more other datastructures that the CCS references, is not a limitation on embodimentsof the invention and any suitable implementation may be used.Additionally, although the example described above indicates thatactions with complex behavior may include instructions in the CCS thatlink to other data structures, it should be appreciated that the CCS mayinclude instructions that link to other data structures for any actionregardless of the complexity of the action to be performed.

As discussed above, voice application 130 may include one or moregrammars and/or vocabularies that are active only for some particularcontexts of a web application loaded in the web browser. The CCS maydefine the particular voice inputs and/or grammars that should be activefor any particular context within the web application, and thisinformation may be provided to voice application 130 in response todetermining an initial context of the web application and when thecontext of the web application changes as determined by agent 116.

In some embodiments, a CCS may define some voice interactions as alwaysbeing active for a particular web application irrespective of theparticular context. For example, a CCS may define voice input such as“Close,” “Exit,” and “Minimize” as always being active for all web pagesin the web application.

In addition to, or instead of, mapping explicit voice commands toactions to perform in the web application using a fixed grammar, someembodiments may employ a CCS that uses free-form input to enable theuser to interact with the web application without having to know theexact voice command mappings that the user must speak to invoke one ormore commands or perform one or more actions. In such embodiments, thevoice application may recognize whatever the user is saying using, forexample, a natural language understanding (NLU) model, and the voiceapplication will attempt to infer an action the user wants to performbased on the NLU result.

Some embodiments may include a combination of a fixed command grammarfor certain actions and free-form input for others. For example, thefixed command “Send message,” when recognized may send an email message,whereas a free-form or dictation grammar may be used to allow a user todictate into an editable text field on a web page. Other embodiments maysupport an action that mixes fixed and free form input, such as a usersaying “Send Mom and email that I'll be an hour late” and compose andsend the requested email.

Additionally, in some embodiments a single voice input may be mapped toperforming multiple actions in accordance with information in a CCS. Forexample, a user may say “Send email to Tom Smith, subject dinner,” andthe response to this input may be to cause the address and subjectfields in an email application to be filled in response to receiving thevoice input. This may be accomplished, for example, by looking up thecorresponding email address for the desired recipient (i.e., Tom Smith)in an address book maintained for a user by the web application todetermine if a corresponding contact is in the address book. If therecipient is found in the address book, the address for the recipientmay be entered in the address field. Similarly, with the subject field,the CCS may specify the voice input for filling in the subject field asfollows: subject <dictation>, where <dictation> refers to free-forminput after the command word “subject.”

In some embodiments, a browser may have multiple web applications openat the same time (e.g., Gmail™ and Facebook®) and each web applicationmay be associated with a different agent that is launched in associationwith the instance of the browser associated with the web applications.In some embodiments, when the web applications are executing indifferent browser windows, the context for determining which voiceinteractions should be active may be determined based on which browserwindow has the current focus. When the current focus is changed (e.g.,by selection of a different window), the set of voice interactions thatare active may also change due to a new window having the focus.

In other embodiments, when multiple browser windows are open, the set ofvoice interactions that are active may be determined in some way otherthan being based on which window currently has the focus. For example,the set of active voice interactions may include all voice interactionsfor all windows that are open regardless of whether they are have thefocus or not, as embodiments of the invention are not limited in theparticular manner in which the set of voice interactions is determinedwhen multiple applications are open.

In some embodiments, a single web page may have multiple webapplications executing thereon. In such embodiments, each of the webapplications may be associated with a particular frame (or sub-frame) onthe web page. An example of this type of scenario is illustrated in FIG.5, which shows a web page 500 loaded by a browser. Web page 500 includesa plurality of fields including toolbar 520 that may be associated witha first web application (e.g., Google Search), an analytics frame thatmay be associated with a second web application (e.g., GoogleAnalytics), an advertising frame 524 that serves up ads from particularvendors, and a map frame 526, which displays a map provided by anotherweb application (e.g., Google Maps).

Some embodiments are directed to determining a collective set ofsupported voice interactions for a web page including multiple frames,wherein at least two of the frames are associated with different webapplications, and wherein each of the at least two frames that areassociated with different web applications have CCSs available to allowa user to use voice input to perform an action in one or more of theframes with which voice input is available. In some embodiments, each ofthe frames may be associated with a different agent. That is, a firstagent associated with a first frame may be configured to monitor eventsin the first frame to determine a context for the first frame and asecond agent associated with a second frame may be configured to monitorevents in the second frame to determine a context for the second frame.

FIG. 6 shows an illustrative process for determining a collective set ofsupported voice interactions in accordance with some embodiments. In act610, based, at least in part, on the context associated with the firstframe, a first set of supported voice interactions may be determinedfrom the CCS associated with the web application executing in the firstframe. For example, if the web application executing in the first frameis Google™ Search and a CCS for Google™ Search is available and has beenaccessed by the first agent, the first set of supported voiceinteractions may be determined based on the CCS for Google™ Search inresponse to determining the current context of the web application inthe first frame.

The process then proceeds to act 620, where a second set of voiceinteractions is determined for the web application executing in thesecond frame. For example, if the web application executing in thesecond frame is Google™ Maps and a CCS for Google™ Maps is available andhas been accessed by the second agent, the second set of supported voiceinteractions may be determined based on the CCS for Google™ Maps inresponse to determining the current context of the web application inthe second frame.

The process then proceeds to act 630, where a collective set ofsupported voice interactions for the web page is determined. Thecollective set of supported voice interactions may be determined in anysuitable way, as embodiments of the invention are not limited in thisrespect. For example, in some embodiments the collective set ofsupported voice interactions may include all (i.e., the union) of thevoice interactions for the contexts in the first set and the second set.In other embodiments, the collective set of supported voice interactionsmay include less than all of the voice interactions in the combinedfirst set and second set.

In response to determining the collective set of supported voiceinteractions for the web page, the process proceeds to act 640, wherethe speech engine is instructed to recognize the collective set of voiceinteractions. For example, if the web page includes two frames with theweb applications Google™ Search and Google™ Maps respectively executingtherein, as described in the examples above, the set of collectivesupported voice interactions may include voice commands such as“Search,” and “Find,” for which the CCS for Google™ Search may defineparticular actions in the Google™ Search frame, and the set ofcollective supported voice interactions may also include voice commandssuch as “Pan left” and “Zoom,” for which the CCS for Google™ Maps maydefine particular actions in the Google Maps frame. Although thediscussion below refers to a CCS for storing information about mappingvoice commands to functionality in a web page, it should be appreciatedthat any data structure may alternatively be used, as embodiments of theinvention are not limited in this respect.

As should be appreciated from the foregoing, the creation of a datastructure (e.g., CCS) to map natural speech to control aspects of webapplications is very flexible, as any desired mapping may be describedin the data structure (e.g., CCS). The inventors have also recognizedand appreciated that web applications are frequently updated with newfunctionality and voice enabling the new functionality usingconventional methods (e.g., by requiring a new version of the voiceapplication) is quite cumbersome. An advantage of some embodiments ofthe invention that use relatively simple data structures such as CCSs isthat updates in functionality to a web application are easy to handle,merely by modifying an existing CCS for the web application to includevoice instructions for the new functionality or, in the case ofsubstantial changes to the web application, writing a new CCS for theweb application that replaces the previous one. By making the CCSspublicly available, updates to voice enable new functionality forupdated web applications can be rapidly disseminated without having toupdate the voice application on any of the client computers. Forexample, the next time a user loads a particular web application intheir browser, some embodiments will automatically detect that a newversion of the CCS is available and will access (e.g., download) the newCCS for use during the browsing session. Moreover, all of these stepsfor updating a CCS may be performed without informing the user that theunderlying CCS for the web application has changed so the process istransparent to the end user.

To enable the centralized creation and updating of CCSs, someembodiments are directed to a web-accessible repository of CCS(s) eachof which specifies a set of voice interactions for at least one webapplication. By maintaining control over the set of CCSs that are usedin accordance with some embodiments of the invention, the validity andsecurity of such information may be better ensured than if users had thecapability to write their own CCS for a web application. The repositoryof CCSs may be associated with one or more server computers configuredto receive requests for CCSs from one or more client computers. The oneor more server computers may include a communication interfaceconfigured to receive a request for a CCS. The request may include theidentity of a web application determined, for example, by an agentexecuting in a web browser of the client computer, as described above.

FIG. 7 shows an illustrative process for providing a CCS from aweb-accessible repository of CCS(s) in accordance with some embodiments.In act 710, a request for voice interaction information for a webapplication is received by the one or more servers associated with theweb-accessible CCS repository. In response to receiving the request, theprocess proceeds to act 720, where it is determined whether the webapplication identified in the request has a corresponding CCS that hasbeen defined for use with the web application. This may be determined inany suitable way. For example, the repository may include a document(e.g., a manifest), which lists the identity (and optionally theversions) of each of the web applications for which a CCS has beencreated and is available.

If it is determined in act 720 that a CCS for the web applicationidentified in the request is available, the process proceeds to act 730,where the requested CCS is provided to the client computer that issuedthe request. In some embodiments, prior to issuing the request to aremotely-located server computer, the client computer may determine ifthe client computer already has a local copy of the current version ofthe CCS, and if a local copy is found, a request to the remotely-locatedserver may not be sent. This determination may be made in any suitableway. For example, prior to issuing a request to the web-accessiblerepository, the agent executing in the browser of the client computermay check local storage to determine if the required CCS is alreadystored locally. If a locally stored copy of the CCS is found, it may bedetermined whether the locally stored copy is the most recent version.In some embodiments, a data structure (e.g., a file or document)including information about the current versions of CCSs available maybe periodically downloaded to the client computer to make thisdetermination.

An illustrative implementation of a client computer system 800 that maybe used in connection with any of the embodiments of the inventiondescribed herein is shown in FIG. 8. The computer system 800 may includeone or more processors 810 and one or more computer-readablenon-transitory storage media (e.g., memory 820 and one or morenon-volatile storage media 830). The processor 810 may control writingdata to and reading data from the memory 820 and the non-volatilestorage device 830 in any suitable manner, as the aspects of the presentinvention described herein are not limited in this respect. To performany of the functionality described herein, the processor 810 may executeone or more instructions stored in one or more computer-readable storagemedia (e.g., the memory 820), which may serve as non-transitorycomputer-readable storage media storing instructions for execution bythe processor 810.

It should be appreciated that client computer 800 is not limited by anyparticular type of computer and may include, but is not limited to, ahandheld computer, a laptop computer, a tablet computer, a desktopcomputer, a smartphone, and any other type of computing device capableof rendering a web application in a web browser.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. It should beappreciated that any component or collection of components that performthe functions described above can be generically considered as one ormore controllers that control the above-discussed functions. The one ormore controllers can be implemented in numerous ways, such as withdedicated hardware, or with general purpose hardware (e.g., one or moreprocessors) that is programmed using microcode or software to performthe functions recited above.

In this respect, it should be appreciated that one implementation of theembodiments of the present invention comprises at least onenon-transitory computer-readable storage medium (e.g., a computermemory, a floppy disk, a compact disk, a tape, etc.) encoded with acomputer program (i.e., a plurality of instructions), which, whenexecuted on a processor, performs the above-discussed functions of theembodiments of the present invention. The computer-readable storagemedium can be transportable such that the program stored thereon can beloaded onto any computer resource to implement the aspects of thepresent invention discussed herein. In addition, it should beappreciated that the reference to a computer program which, whenexecuted, performs the above-discussed functions, is not limited to anapplication program running on a host computer. Rather, the termcomputer program is used herein in a generic sense to reference any typeof computer code (e.g., software or microcode) that can be employed toprogram a processor to implement the above-discussed aspects of thepresent invention.

Various aspects of the present invention may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and are therefore notlimited in their application to the details and arrangement ofcomponents set forth in the foregoing description or illustrated in thedrawings. For example, aspects described in one embodiment may becombined in any manner with aspects described in other embodiments.

Also, embodiments of the invention may be implemented as one or moremethods, of which an example has been provided. The acts performed aspart of the method(s) may be ordered in any suitable way. Accordingly,embodiments may be constructed in which acts are performed in an orderdifferent than illustrated, which may include performing some actssimultaneously, even though shown as sequential acts in illustrativeembodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed. Such terms areused merely as labels to distinguish one claim element having a certainname from another element having a same name (but for use of the ordinalterm).

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing”, “involving”, andvariations thereof, is meant to encompass the items listed thereafterand additional items.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

What is claimed is:
 1. A method of determining a collective set ofsupported voice interactions for a plurality of frames of a web pagedisplayed in a window of a web browser, wherein each of the plurality offrames includes content for a different web application, wherein thecontent for each of the plurality of frames is displayed simultaneouslyin the window of the web browser, wherein the plurality of framesincludes a first frame and a second frame, wherein the first framedisplays content for a first web application rendered by the web browserand the second frame displays content for a second web applicationrendered by the web browser, wherein the first web application isdifferent from the second web application, the method comprising:identifying a first data structure that includes information identifyinga plurality of contexts of the first web application and supported voiceinteractions for the first web application in each of the plurality ofcontexts of the first web application; determining a first currentcontext of the first web application, wherein determining the firstcurrent context comprises analyzing whether a particular marker ispresent in the content displayed in the first frame; determining based,at least in part, on the first current context of the first webapplication and the information included in the first data structure, afirst set of supported voice interactions available for the first frame;identifying a second data structure that includes informationidentifying a plurality of contexts of the second web application andsupported voice interactions for the second web application in each ofthe plurality of contexts of the second web application; determiningbased, at least in part, on a second current context of the second webapplication and the information included in the second data structure, asecond set of supported voice interactions available for the secondframe; determining the collective set of supported voice interactionsbased on the first set of supported voice interactions and the secondset of voice interactions; and instructing an external speech engine torecognize voice input corresponding to the collective set of voiceinteractions.
 2. The method of claim 1, further comprising: associatinga first agent with the first frame and a second agent with the secondframe, wherein the first agent is configured to monitor first eventsassociated with the first frame and the second agent is configured tomonitor second events associated with the second frame; and wherein thefirst current context of the first web application is determined based,at least in part, on the first events and the second current context ofthe second web application is determined based, at least in part on thesecond events.
 3. The method of claim 1, wherein the collective set ofsupported voice interactions comprises a union of the supported voiceinteractions in the first set and the second set.
 4. The method of claim1, wherein the collective set of supported voice interactions includesless than all of the supported voice interactions in the first set andthe second set.
 5. The method of claim 1, wherein determining the firstset of supported voice interactions comprises determining an identity ofthe first web application.
 6. The method of claim 5, wherein determiningthe identity of the first web application is performed based, at leastin part, on an identifier for a web page of the first web application.7. The method of claim 1, wherein determining the first set of supportedvoice interactions available for the first frame comprises monitoringfor browser events in the first frame.
 8. A non-transitorycomputer-readable storage medium encoded with a plurality ofinstructions that, when executed by a computer, perform a method ofdetermining a collective set of supported voice interactions for aplurality of frames of a web page displayed in a window of a webbrowser, wherein each of the plurality of frames includes content for adifferent web application, wherein the content for each of the pluralityof frames is displayed simultaneously in the window of the web browser,wherein the plurality of frames includes a first frame and a secondframe, wherein the first frame displays content for a first webapplication rendered by the web browser and the second frame displayscontent for a second web application rendered by the web browser,wherein the first web application is different from the second webapplication, the method comprising: identifying a first data structurethat includes information identifying a plurality of contexts of thefirst web application and supported voice interactions for the first webapplication in each of the plurality of contexts of the first webapplication; determining a first current context of the first webapplication, wherein determining the first current context comprisesanalyzing whether a particular marker is present in the contentdisplayed in the first frame; determining based, at least in part, onthe first current context of the first web application and theinformation included in the first data structure, a first set ofsupported voice interactions available for the first frame; identifyinga second data structure that includes information identifying aplurality of contexts of the second web application and supported voiceinteractions for the second web application in each of the plurality ofcontexts of the second web application; determining based, at least inpart, on a second current context of the second web application and theinformation included in the second data structure, a second set ofsupported voice interactions available for the second frame; determiningthe collective set of supported voice interactions based on the firstset of supported voice interactions and the second set of voiceinteractions; and instructing an external speech engine to recognizevoice input corresponding to the collective set of voice interactions.9. The computer-readable storage medium of claim 8, wherein the methodfurther comprises: associating a first agent with the first frame and asecond agent with the second frame, wherein the first agent isconfigured to monitor first events associated with the first frame andthe second agent is configured to monitor second events associated withthe second frame; and wherein the first current context of the first webapplication is determined based, at least in part, on the first eventsand the second current context of the second web application isdetermined based, at least in part on the second events.
 10. Thecomputer-readable storage medium of claim 8, wherein the collective setof supported voice interactions comprises a union of the supported voiceinteractions in the first set and the second set.
 11. Thecomputer-readable storage medium of claim 8, wherein the collective setof supported voice interactions includes less than all of the supportedvoice interactions in the first set and the second set.
 12. Thecomputer-readable storage medium of claim 8, wherein determining thefirst set of supported voice interactions comprises determining anidentity of the first web application.
 13. The computer-readable storagemedium of claim 12, wherein determining the identity of the first webapplication is performed based, at least in part, on an identifier for aweb page of the first web application.
 14. The computer-readable storagemedium of claim 8, wherein determining the first set of supported voiceinteractions available for the first frame comprises monitoring forbrowser events in the first frame.
 15. A computer for determining acollective set of supported voice interactions for a plurality of framesof a web page displayed in a window of a web browser, wherein each ofthe plurality of frames includes content for a different webapplication, wherein the content for each of the plurality of frames isdisplayed simultaneously in the window of the web browser, wherein theplurality of frames includes a first frame and a second frame, whereinthe first frame displays content for a first web application rendered bythe web browser and the second frame displays content for a second webapplication rendered by the web browser, wherein the first webapplication is different from the second web application, the computercomprising: at least one processor programmed to: identify a first datastructure that includes information identifying a plurality of contextsof the first web application and supported voice interactions for thefirst web application in each of the plurality of contexts of the firstweb application; determine a first current context of the first webapplication, wherein determining the first current context comprisesanalyzing whether a particular marker is present in the contentdisplayed in the first frame; determine based, at least in part, on thefirst current context of the first web application and the informationincluded in the first data structure, a first set of supported voiceinteractions available for the first frame; identify a second datastructure that includes information identifying a plurality of contextsof the second web application and supported voice interactions for thesecond web application in each of the plurality of contexts of thesecond web application; determine based, at least in part, on a secondcurrent context of the second web application and the informationincluded in the second data structure, a second set of supported voiceinteractions available for the second frame; determine the collectiveset of supported voice interactions based on the first set of supportedvoice interactions and the second set of voice interactions; andinstruct an external speech engine to recognize voice inputcorresponding to the collective set of voice interactions.
 16. Thecomputer of claim 15, wherein the at least one processor is furtherprogrammed to: associate a first agent with the first frame and a secondagent with the second frame, wherein the first agent is configured tomonitor first events associated with the first frame and the secondagent is configured to monitor second events associated with the secondframe; and wherein the first current context of the first webapplication is determined based, at least in part, on the first eventsand the second current context of the second web application isdetermined based, at least in part on the second events.
 17. Thecomputer of claim 15, wherein the collective set of supported voiceinteractions comprises a union of the supported voice interactions inthe first set and the second set.
 18. The computer of claim 15, whereinthe collective set of supported voice interactions includes less thanall of the supported voice interactions in the first set and the secondset.
 19. The computer of claim 15, wherein determining the first set ofsupported voice interactions comprises determining an identity of thefirst web application.
 20. The computer of claim 19, wherein determiningthe identity of the first web application is performed based, at leastin part, on an identifier for a web page of the first web application.